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Examining the Relationship between LA’s BEST Program Attendance and Academic 

Achievement of LA’s BEST Students 

Denise Huang, Seth Leon, Deborah La Torre & Sima Mostafavi 
CRESST/University of California, Los Angeles 

Abstract 

Researchers and policymakers are increasingly interested in the impact of afterschool 
programs on youth development. Even though numerous studies have investigated the 
impact of afterschool participation on academic outcomes, there is limited research on 
the differential impact of afterschool programs based on students’ participation rate. This 
study bridges that research gap and presents results from a study of the effectiveness of 
the LA’s BEST afterschool program based on different levels of student participation. 

This research tracked 4 years of the academic histories for two cohorts of students 
participating in LA’s BEST. We separated the students in each cohort into four categories 
based on their intensity of attendance in LA’s BEST and then used a propensity based 
weighting method to remove existing differences in student background characteristics. 
Hierarchical growth modeling was employed to analyze the academic outcomes. Results 
indicate that math achievement outcomes of students vary by intensity of program 
participation. Student participants who attended LA’s BEST over 100 days per year 
demonstrated greater math achievement growth than students with low program 
attendance. This finding was consistent, and was statistically significant, for both cohorts 
of students. In contrast, although the trend for English-language arts achievement growth 
was positive, and followed a developmental pattern similar to math, it did not vary 
significantly by intensity of program participation. This finding was also consistent for 
both cohorts of students. 



Chapter I: 

Introduction 

In recent years, interest and funding in afterschool programs has increased 
significantly. For example, California increased its yearly budget for afterschool programs 
from 120 to 550 million during the 2006-07 fiscal year (California AfterSchool Network, 
2007).' As a result, funders and policymakers are demanding greater accountability of 
programs. 

Ever since the enactment of No Child Left Behind Act in 2001 (NCLB; 2002), 
achievement gains resulting from afterschool participation have been of particular interest 
(Lauer, et ah, 2006; Miller 2003). However, findings have been inconsistent (Fashola, 1998; 



* As mandated by Proposition 49, funding for afterschool programs was increased once the California state 
budget reached a level making the release of funds feasible (California AfterSchool Network, 2007). 
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Vanderhaar & Munoz, 2006). The challenge for researchers is partly due to the wide 
variation of program goals, difficulty in obtaining valid control groups, the inherent potential 
of selection bias in the afterschool population, difficulty in obtaining clean records of data, 
the high transience rates of the students, and in particular, the failure to differentiate among 
the dosage^ students receive (Lauer et ah, 2003). 

For any intervention project, it is necessary for subjects to receive adequate treatment in 
order to demonstrate effects. However, afterschool studies rarely examine the importance of 
dosage. When conducting program evaluations, it is very common for studies to group all 
participants (regardless of attendance days) as the treatment group and the non-participants 
(without proper control of pre-existing differences) as the comparison group. Thus, although 
inconsistent results can stem from many factors, such as those previously mentioned, failing 
to consider the “dosage” effect is one of the most important. Whereas some students might 
have regular attendance at an afterschool program, others might “drop-in” as needed. In cases 
of inconsistent attendance, it is unrealistic to expect significant academic gains. 

The goal of this study is to examine the long-term relationship between participation in 
LA’s BEST and academic achievement. Accordingly, the main research question for this 
study is as follows: 

• Do the achievement outcomes of LA’s BEST students’ vary as a function of their 
different intensity levels of afterschool participation? 

Only recently have researchers started examining the effects of dosage level on the 
academic outcomes of afterschool programs. These studies have found that students who 
attend afterschool programs more and experience more exposure, benefit more from the 
program (Lauer et ah, 2003; McComb & Scott-Little, 2003; Frankel & Daley, 2007). The 
purpose of this study is to further this research by comparing students with different dosage 
levels using propensity score matching to reduce self-selection bias and to examine the 
students’ achievement trends over a period of 4 years. 

Chapter II: 

Review of Literature 

What academic outcomes are associated with afterschool program participation? When 
reviewing research on the academic impact of afterschool participation, results are mixed. It 
is not unusual to come across null (Lauver, 2002; Dynarski, et ah, 2003; Vanderhaar & 



^ Feister, Simpkins and Bouffard (2005), define dosage as a measure of attendance intensity that focuses on the 
amount of time a participant attends a program within a specified period (e.g., hours per week, days per month, 
days in a year, etc.). 



2 




Munoz, 2006) and positive (Redd, Coehran, Hair & Moore, 2002; Dynarski, et al., 2003) 
results. 

Studies of afterschool programs have reported on an array of positive academic 
outcomes. Bergin, Hudson, Chryst & Resetar (1992) found positive associations between 
afterschool participation and higher achievement scores. Their study followed a group of 
kindergartners who attended an afterschool program and compared them to a control group. 
Initially, the standardized test scores of both groups were below national average. However, 
by the spring of first grade, the treatment group was outperforming the control group and was 
performing above national norms. Afterschool participation is also associated with higher 
classroom grades, higher math and reading scores, increased day school attendance, lower 
dropout rates, higher homework completion rates, and higher graduation rates (Goerge, 
Cusick, Wasserman & Gladden, 2007; Little & Harris, 2003; Sheley, 1984). While at the 
same time, others have reported mixed, insignificant or negative outcomes regarding 
academic performance^, school retention, feelings of safety, and behavior to name a few 
(Cooper, Charlton, Valentine & Muhlenbruck, 2000; Dynarski, et al, 2003; James, 1997; 
Vanderhaar & Munoz, 2006). However, most studies that have evaluated dosage level have 
found positive effects for students who attend more consistently (McComb & Scott-Little, 
2003). The following sections discuss two important considerations in conducting afterschool 
research and evaluations. 

The Dosage Effect 

Dosage effect is a critical factor to examine when assessing the effect of an 
intervention. More specifically, examining dosage helps to determine whether participants 
are receiving sufficient treatment in order to demonstrate effect. Even though dosage 
(defined as intensity of participation from here on) is very important in determining program 
success, recent literature on afterschool programs has only begun to investigate this issue. In 
general, these studies have found a positive relationship between intensity of participation 
and positive student outcomes. For instance, Frankel and Daley (2007) found that higher 
afterschool attendance is associated with higher academic achievement, while Goldschmidt, 
Huang and Chinen (2007), found that medium (10-14 days per month) and high attendance 
(15 or more days per month) in an afterschool program is associated with lower juvenile 
crime rate. In recent years, multiple studies have also found a relationship between 
afterschool attendance intensity and higher day school attendance (Frankel & Daley, 2007; 



^ Some studies looking at both math and reading outcomes have only found effects for math whereas others 
have only found effects for reading 
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Huang, Gribbons, Kim, Lee & Baker, 2000; Welsh, Russell, Williams, Reisner & White, 
2002; Munoz, 2002). 

Speeifieally, in 2007, Frankel and Daley released a report that found an assoeiation 
between high dosage of aftersehool partieipation and higher math assessment seores, 
English-language arts assessment seores, and day sehool attendanee. They ereated four 
attendanee level eategories: 1-20 days, 21-50 days, 51-100 days, and more than 100 days 
per year. They found that, in order to benefit aeademieally, the elementary sehool students 
needed to attend the aftersehool program for at least 100 days per year and middle sehool 
students needed to attend at least 50 days annually. 

Similarly, Jenner and Jenner (2007) examined the impaet of program partieipation 
intensity on aeademie outeomes. They found a linear and positive relationship between 
partieipation level and aeademie outeomes sueh as math, reading, language arts, and seienee 
seores. Their analyses plaeed the minimum attendanee level neeessary for measuring impaet 
at 30 days annually. 

Along the same lines, Munoz (2002) looked at aftersehool program partieipation and 
student outeomes among inner eity students in Louisville, Kentueky. The author established 
two aftersehool program attendanee level eategories using the mean number of visits by all 
partieipants. He found a positive relationship between intensity of aftersehool program 
attendanee and day sehool attendanee. In addition, he found non-signifieant eorrelations 
between higher aftersehool intensity and lower suspensions as well as greater GPA. 

Intensity level of aftersehool attendanee ean also prediet soeial outeomes; for instanee, 
Goldsehmidt, Huang, and Chinen (2007) examined the long-term effeetiveness of aftersehool 
programs in lowering juvenile erime rates. They found that students who eonsistently 
attended LA’s BEST demonstrated a substantive signifieant reduetion in juvenile erime as 
compared to students with inconsistent attendance and no attendance. 

Additionally, a meta-analysis (Lauer et ah, 2003) examining 35 out-of-school time 
(OST) programs"^ for assisting at-risk students in reading and/or math identified the duration 
of OST as a moderator. They found that for both reading and math, effect sizes were larger 
for OST programs that were more than 45 hours annually. Unlike the four studies previously 
mentioned, Lauer and colleagues (2006) looked at program duration instead of students’ 
program attendance. They did this because of incomplete access to attendance data. This 



Out-of-school time refers to activities that children participate in when they are not in school and that are not 
mandated by school attendance. This may include before school, aftersehool, and summer programs (Lauer, et 
ah, 2006 ). 
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study defined program duration as the total number of hours that the program was offered to 
partieipants rather than the number of days students attended. 

Finally, in reviewing researeh on partieipation and outeomes in aftersehool programs, 
MeComb and Soott-Little (2003) eoneluded that students who attend aftersehool programs 
more frequently and for longer periods benefit the most. They suggest that aftersehool 
programs should be an integral part of sehool’s aeademie and developmental programs. They 
stated that in all eases where data was examined using “intensity” level, results favored 
students who had partieipated at higher levels. 

Reducing Selection Bias 

Another frequent eritique of aftersehool studies is seleetion bias (Fashola, 1998; 
Hollister, 2003; Little & Harris, 2003; Seott-Little, Hamann & Jurs, 2002). Beeause 
aftersehool program partieipation is voluntary, students self-seleet themselves into 
partieipation and non-partieipation groups.^ In eomparing partieipating students to non- 
partieipating students in the same sehool, there are inherent biases that researehers need to 
balanee or eontrol in order for the findings to be valid. Furthermore, due to the soeial eontext 
of aftersehool programs, reaehing the “gold standard” of researeh is diffieult. Aeeording to 
the Ameriean Institutes for Researeh (2002), the “gold standard” is researeh that meets all of 
the standards of seientifieally based researeh as ealled for in the NCLB Aet (2002). This 
ineludes the use of experimental designs, ineluding randomization and eontrol groups. In 
reality, it is often diffieult, and potentially unethieal, for most aftersehool programs to 
randomize their partieipants unless the programs are grossly oversubseribed. For, unless 
programs have many more applieants than available spaees, random assignment would mean 
refusing to aeeept some students into the program so that they eould serve as eontrols. 
Students who are refused enrollment may end up unsupervised and without the homework 
help they desperately need. As a result, many studies laek either true experimental eontrol or 
a valid eomparison group. Thus, most studies in this field are quasi-experimental, with 
researehers using a eomparison group and making use of statistieal eontrols. In these quasi- 
experimental studies, one needs to be eautious when inferring eausality. With this in mind, 
the present study reduees self-seleetion bias by removing pre-existing eategory differenees 
using propensity seores. Propensity seores are estimated in order to aeeount for potential 
differenees in student baekground eharaeteristies, sueh as gender and ethnieity. By redueing 
initial differenees aeross different groups, one ean more eonfidently attribute differenees in 
aehievement outeomes to treatment intensity. 



^ Parents may also choose to enroll or “select” their children for participation. 
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In summary, although many researchers indicate that afterschool programs are a 
potentially powerful resource that can help increase student’ academic achievement, the 
reported findings on academic outcomes are mixed. In our brief review of literature, we 
found that many studies that claim positive outcomes reported academic improvement in 
students with a higher dosage of afterschool participation, and those that reported null or 
negative findings more often looked at participants of afterschool as an aggregated group. 
Recently, researchers have begun to examine the relationship between regular afterschool 
participation and academic outcomes. Even as the literature states that quality afterschool 
programs can teach students the academic and social skills that they need to avoid anti-school 
behaviors and contribute to academic resiliency, sufficient exposure to effective afterschool 
environments is necessary in order for students to reap the benefits. At the same time, 
although it is necessary to look at the intensity of participation as a contribution to student 
outcomes, it is also important to reduce the selection bias that is inherent in the field of 
afterschool research in order to add validity to the findings of the studies. This study intends 
to fill the research gap by examining the impact of differential intensity of exposure to 
afterschool programming, specifically LA’s BEST, on student academic achievement and 
using propensity matching as a technique to reduce self-selection bias. First, we provide a 
brief description of the LA’s BEST program. 

The LA’s BEST Program 

Los Angeles Better Educated Students for Tomorrow (LA’s BEST) was first 
implemented in the fall of 1988. The program is under the auspices of the Mayor of Los 
Angeles, the Superintendent of the Los Angeles Unified School District (LAUSD), a board 
of directors, and an advisory board consisting of leaders from business, labor, government, 
education, and the community. 

LA’s BEST seeks to provide a safe haven for at-risk students in neighborhoods where 
gang violence, drugs, and other types of anti-social behaviors are common. The program is 
housed at selected LAUSD elementary schools and is designed for students in kindergarten 
through fifth/sixth grade. The LA’s BEST sites are chosen based on certain criteria, such as 
low academic performance and their location in low-income, high-crime neighborhoods. For 
optimal program success and to ensure buy-in from the principals and the school staff, the 
school principals have to write an official letter of request for the program to be placed in 
their school site. 

LA’s BEST is a free program open to all students in the selected sites on a first come 
first serve basis. Students who sign up for the program are expected to attend 5 days a week 
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in order to reap the full benefits of the program offerings. Currently, LA’s BEST serves a 
student population of approximately 30,000 with about 80% Hispanie and about 12% Blaek 
elementary students. English Eeamers eomprise at least half of the student population from 
most sites. Of this population, the majority’s primary language is Spanish; whereas the other 
pereentage of the English Eeamer population is eomposed of those whose first language is of 
Asian or Paeifie origin. 

Parents often mention homework help and proper supervision as the primary ineentives 
for enrolling their ehildren. Teaehers may also reeommend students for EA’s BEST due to 
behavioral or aeademie needs. Students enjoy the program due to its supportive staff and 
positive environment eondueive for aeademie aehievement and engagement of 
extraeurrieular aetivities. 

Program offerings. Sinee its ineeption in 1988, EA’s BEST has adapted and updated 
their goals in response to edueational polieies, researeh, and theory. Over the years, the 
program has moved past its initial emphasis on providing a safe environment and edueational 
enriehment to an emphasis on the development of the whole-ehild. In developmental theory, 
a whole-ehild eurrieulum is one that eultivates the development of students’ intelleetual, 
soeial, and emotional well-being so that ehildren ean aehieve their full potential (Sehaps, 
2006; Hodgkinson, 2006). At EA’s BEST, their 3% beats foeus on the whole-ehild by 
emphasizing students’ intelleetual, soeial-emotional, and physieal development. 

Cognitive beat & Homework beat 

Intelleetual development sueh as: 

• Responsibility and positive work habits - through emphasis on the importanee of 
completing assignments, teaching learning strategies and study skills, and providing 
a learning climate that enforces positive attitudes towards school. 

• Love of learning - through active participation, explorations, and engaging 
research-based activities. 

• Self-efficacy - through guided experiences, challenging activities, and relationship 
building between staff and students. 

• Future aspirations - through high expectations, activities that build self-reliance, 
value of education, collaborations, and critical thinking. 

Recreational beat. 

Physical and social-emotional development such as: 

• Sense of safety & security - through providing students with a safe and nurturing 
environment. 
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• Healthy lifestyle - through eurrieulum and aetivities that promotes drug and gang 
prevention, healthy eating habits, and plenty of exereise. 

• Soeial eompetenee - through demonstrating and enhaneing students’ respeet for self 
and others, and providing students with opportunities to form friendships and 
develop trust and respeet with peers and adults. 

• Sense of eommunity - through providing students with opportunities to partieipate 
in eommunity-sponsored events, volunteer in eommunity assignments, and offering 
field trips to loeal business and organizations. 

• Respeet for diversity - through role modeling and eurrieulum that enhanees 
awareness and responsibility to eaeh other within their diverse eommunity. 

To summarize, the mission of LA’s BEST is to provide engaging settings so that: eaeh 
student learns in an intelleetually ehallenging environment that is physieally and emotionally 
safe for both students and adults; eaeh student ean be aetively engaged in learning aetivities 
that are eonneeted to their sehool and broader eommunity; And most importantly, eaeh 
student has aeeess to extra-currieular aetivities, aeademie enhaneements, and qualified, 
caring adults. 

Because the central theme of the LA’s BEST mission is to empower both staff and 
student members, and to build on students’ daily life experiences with program offerings; the 
organization gives each site autonomy to structure their own program, as long as the site 
coordinator and staff adhere to the foundational principles of LA’s BEST.^ As a result, each 
site has distinct characteristics and program themes (such as arts, self-esteem, conflict 
resolution, technology, etc). Subsequently, relationships with the day school, levels of 
school^ and community supports also tend to vary with each site (see Huang, et ah, 2006). 

The following list provides an overview of the different educational and enrichment 
activities offered: 

• Cognitive/ Academic - This includes homework time, tutoring, academic incentive 
programs, math and science activities, reading and writing activities, computer 
activities, and psychological programs addressing conflict resolution skills. 

• Recreational - This includes arts and crafts, cooking, games, holiday activities, and 
sports such as aerobics, karate, and team sports. 



^ The snack and homework periods are the common components of all LA’s BEST sites. The education and 
enrichment sessions are grounded on the principles of being: (a) cognitive/ academic (activities in school 
subject matter; (b) recreational (physical fitness); and (c) part of the performing arts (i.e. dance, drama, etc.). 

^ In a qualitative study of six LA’s BEST sites, Huang and colleagues (2006) found that most principals had a 
cooperative working relationship with LA’s BEST site staff. 




• Performing and Visual Arts - This includes choir and music, dance, drama/theater, 
flag/drill team, museum visits, art camps, etc. 

• Health and Nutrition - This includes the study of nutrition and healthy habits, 
exercise programs such as tennis and skating, and the BEST Fit community health 
fair. 

• Community and Culture - This includes community programs, such as adopt-a- 
grandparent, and community days; and cultural programs, such as those dedicated 
to Black history, “Folklorico,” and other cultural holiday celebrations. 

• Parental involvement activities - These fall under four categories: 

> Celebrations, such as Halloween Kidfest, Community Jam, Awards Days. 

> Programs for children, including parent volunteers for daily activities, parent 
attendance of field trips. 

> Programs for parents, including parent workshops, guest speakers for parental 
education. 

> Communications/information, including open house events, assemblies, 
parent-teacher meetings. 

The educational and enrichment activities mainly come from three different sources: (a) 
curricula purchased from education vendors, such as KidzFit^ and KidzMath^; (b) activities 
developed by the education and staff development departments at FA’s BEST operations; 
and, (c) activities designed by the site staff.'*’ 

Quality assurance. For continuous improvements, FA’s BEST employs both internal 
and external evaluators. Their operations office includes both a Director of Evaluation and 
research analysts. The internal evaluation team conducts regular meetings with field staff to 
provide a forum for sharing experiences and examples of what works and what does not 
work with staff and administrators at the operation office. External evaluations often involve 
feedback from staff, day school teachers, students and parents; they gauge the short and or 
long-term effects of specific program components, or overall program effects. 

Results from evaluations are discussed at site coordinator meetings, and are used to 
determine whether individual sites and the program are meeting goals and objectives. 



“ Afterschool KidzLit is an enrichment program that emphasizes literacy skills, written expression, core values, 
connections, and thinking skills by having children read and talk about books. The program is research based 
and is aligned with the National Council of Teachers of English (NCTE) standards. 

^ Afterschool KidzMath is an enrichment program that emphasizes the enjoyment and development of math 
skills. Lessons are structured around the use of math games and math-themed children’s books. 

Site staff members receive support from the program coaches and their site coordinators in developing and/or 
implementing activities. 
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Chapter III: 

Study Design and Methods 

Since the formation of LA’s BEST in 1988, the National Center for Research on 
Evaluation, Standards, & Student Testing (CRESST) has been conducting evaluations of the 
program. As a result, CRESST has established a longitudinal database on these students. The 
longitudinal database includes student demographics and academic information such as 
student achievement scores on English-language arts and mathematics standardized tests. 

The basis for this study sample is comprised of the EAUSD student database that 
CRESST has collected and stored since the 1992-93 school year. The first step in building a 
sample consists of generating a sampling frame. We accomplished this task by going back 
through the historical records and tracking all available information for all students from the 
2002-03 school year through the 2006-07 school year. 

The following describes the study design and the data analysis strategies for this study. 

Study Design 

This study employs a quasi-experimental design that consists of a longitudinal sample 
of both academic and EA’s BEST program attendance data. The sample is comprised of 
roughly 10,000 students from EA’s BEST programs. The sample includes two cohorts of 
students with base years in 2002-03 and 2003-04. We separated students in each cohort into 
four categories based on their intensity of attendance in the EA’s BEST program. We also 
employed a propensity based weighting method to minimize existing differences m student 
background characteristics across the four EA’s BEST program attendance categories. Once 
this was completed, we took advantage of this panel structure and applied hierarchical 
growth modeling to academic outcomes. This method allowed us to examine students’ 
academic growth while controlling for student and school-level background characteristics. 
Given that we had student background information, we also examined moderating factors 
such as gender, race/ethnicity, language proficiency, and socioeconomic status. 

Data Analysis Methods 

We utilized the longitudinal nature of the data and followed academic data over time. 
The benefit of this longitudinal structure is twofold. First, it allows us to move beyond 
traditional pre-post analysis, which is limited by data requirements and explanatory 
possibilities (Rogosa, Brandt, & Zimowski, 1982; Raudenbush & Bryk, 2002). We employed 
growth-modeling techniques that examined individual trajectories (Rogosa et ah, 1982) and 
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have more flexible data requirements.'' Seeond, we separated initial status from growth, thus 
avoiding spurious negative eorrelations between where students start and their ensuing 
growth (Bloomquist, 1977). 

Propensity seores were estimated to aeeount for potential differenees in student 
baekground eharaeteristies. These seores are eomputed from a large reservoir of potential 
controls by applying a systematic weighting procedure. In other words, the propensity score 
is the conditional probability of being assigned to the treatment condition given a set of 
observed covariates. It is commonly estimated using a logistic link function. Because we 
have four comparison groups ordered by attendance intensity, rather than a single treatment 
and a single control, we created a propensity scalar that corresponds with the ordered 
likelihood of belonging to one of the four intensity groups. We estimated this propensity 
scalar using ordinal logistic regression. 

In order to examine the effects of LA’s BEST on achievement and achievement growth, 
we employed a hierarchical linear model (HEM) design that has the advantages of directly 
modeling growth trajectories and being more flexible than traditional analyses. Because 
observations are nested within individuals, time intervals need not be constant across 
individuals as in traditional repeated measures analyses (Raudenbush & Bryk, 2002), and the 
number of observations per person may vary. Thus, this HEM design allows flexible 
specification of the covariance structure at every level of the analysis for this study (Snijders 
& Bosker, 1999). 

The HEM analysis is based on a three-level model. Two separate models were 
conducted for each cohort, one for math and one for English-language arts. In these models. 
Level 1 represents time nested within students. There are four time points for each 
achievement model, with achievement at each time point serving as the outcome. Before 
specifying the growth models, we examined the overall achievement growth patterns to 
determine whether a quadratic or logarithmic transformation would provide a better fit than a 
simple linear model. Because neither transformation resulted in an improved fit, we modeled 
linear growth. 

At Level 1 we model achievement to be predicted by time (school year). The Level 1 
model has two coefficients for each child including an intercept and a slope. The intercept for 
this level is initialized at zero for the first time point. Level 2 accounts for student-level 
effects. At this level, the achievement intercept and the achievement slope over time are 



* * such as not requiring balanced data (Raudenbush & Bryk, 2002) and managing missing data due to attrition 
(Hox, 2002) 
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modeled as funetions of LA’s BEST attendanee intensity and day sehool attendanee. At 
Level 3, information regarding mean aehievement at the sehool level is ineluded in the 
model. Only the intereept and slope are allowed to vary randomly over this level. The slopes, 
due to the effeet of LA’s BEST attendanee intensity and day sehool attendanee, are assumed 
eonstant at this level of the model. 

This model is performed on the weighted sample in whieh differenees in baekground 
eharaeteristies and the initial aehievement outeomes aeross intensity levels have been 
removed. The primary relationship of interest is that between attendanee intensity and the 
slope of aehievement growth over time. The presenee of a signifieant relationship between 
attendanee intensity and the slope of aehievement growth over time, after eontrolling for day 
sehool attendanee and other baekground eharaeteristies, would provide evidenee of the EA’s 
BEST intensity of program attendanee impaet. 

Defining the Study Sample 

The basis for the sample is eomprised of the EAUSD dataset that CRESST has 
colleeted and stored sinee the 1992-93 sehool year. The first step in building a sample 
eonsists of generating a sampling frame. We aeeomplished this task by going baek through 
the historieal reeords and traeking 4 years of baekground and California Standards Tests 
(CSTs) aehievement data for the students in the two eohorts. The seeond- and third-grade 
cohorts were selected because 3 years of complete EA’s BEST attendance data and 4 years of 
complete background data (achievement scores, day school attendance, etc.) was available 
for students in these cohorts. The following describes how we defined the two cohorts. 

Grade 3 cohort (2003-04). Four years of achievement results were available for this 
cohort spanning the 2002-03 school year through the 2005-06 school year. Only students 
with valid CST achievement scores and EA’s BEST attendance days reported during the 
study period were analyzed. Subsequently, students who were in third grade in 2003-04 were 
followed from 2002-03 to their projected fifth-grade year in 2005-06.'^ Because we 
employed HEM analysis to control for school-level effects, a minimum of 10 LA’s BEST 
students per school was required for admission into the study sample. The resulting samples 
included 4,03 1 students in the math sample and 4,060 students in the English-language arts 
sample from 112 schools. 

Grade 2 cohort (2003-04). Four years of achievement results and EA’s BEST 
attendance data were available for this cohort spanning the 2003-04 school year through the 



Students may have been retained following the 2003-04 school year. 
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2006-07 school year. As with the first cohort, we only included students with valid CSX 
achievement scores and LA’s BEST attendance during the study period. We followed 
students who were in second grade in 2003-04 through 2006-07, their projected year in fifth 
grade barring retention. Furthermore, because we employed HEM analysis to control for 
school-level effects, a minimum of 10 EA’s BEST students per school was required for 
admission into the sample. The resulting sample included 5,995 students in the math sample 
and 5,991 students in the English-language arts sample from 134 schools.'^ 

Defining Attendance Intensity 

Examination of student attendance patterns indicates that students participate in EA’s 
BEST with varying regularity. Therefore, it is neeessary to set eriterion to measure the 
intensity of attendanee. In order to aeeomplish this, we eomputed the average attendanee of 
all students in EA’s BEST over the 3 study years and then eategorized attendanee into four 
levels of intensity. In order to expand on the work of Frankel & Daley (2007), we defined the 
four intensity levels with the same eut points as used in their study. For the Grade 3 eohort, 
attendanee intensity was based on the period from 2003-04 to 2005-06. For the Grade 2 
eohort, attendanee intensity was based on the period from 2004-05 to 2006-07. As with 
Frankel & Daley we did not expeet students who average less than 20 days of attendanee to 
benefit from the program; therefore, we elassified them as Eevel 1. We elassified students 
attending 21-50 days on average as Eevel 2, and those attending 51-100 days as Eevel 3. We 
defined regular attendanee (Eevel 4) as those students who averaged greater than 100 days of 
LA’s BEST attendanee per year. 

Controlling for Existing Population Differences 

Because we did not randomly assign students to the four intensity levels, it was 
necessary to control for existing differences in student background characteristics so that 
causal interpretations could be explored. In social science, randomized controlled 
experiments are often difficult to achieve due to study design and or ethical issues; 
subsequently quasi-experimental designs using propensity scoring methods are gaining 
widespread use. Typically, these designs employ logistic regression to estimate the 
probability that a subject is in a treatment group compared to a control group, and then use 
the propensity outcome to create balance among the student background characteristics. This 
process can be done using matching, stratum, or weighting techniques. In this study, we 
created an ordered treatment variable with four levels rather than a simple dichotomous 



The Grade 2 cohort includes larger samples of students and schools due to the inclusion of students from new 
school sites added to LA’s BEST in the 2006-07 year. 
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treatment eompared to a eontrol. In the literature, adaptations to the basie propensity seoring 
method have been proposed in the ease of an ordinal or dosage based treatment variable. 
Therefore, we adopt this approaeh by using ordinal logistie regression within a hierarehieal 
linear modeling (HLM) framework to ereate a single propensity sealar. 

Step 1 - HLM ordinal logistic regression. We employed ordinal logistie regression 
within an HLM framework to model the relationship between student baekground 
charaeteristies and the likelihood of a student attending the LA’s BEST program at the 
varying intensity levels. Level 1 (student level) indieators for baseline aehievement, day 
sehool attendanee, parental edueation, ethnieity (% Hispanie and % Blaek), Gender (% 
female). Limited English Profieient (LEP) and Initially Fluent English Profieient (IFEP) 
status were entered with the four level ordinal intensity variable used as the outeome. Eaeh 
sehool represents a Level 2 unit and the average sehool aehievement effeet on the student- 
level intereept was ineluded in the model. We then transformed the model eoeffieients to 
create a single propensity scalar, after which we divided the propensity scalar into quintiles. 
In other words, we gave each student a score of 1-5 based on his or her propensity score. The 
creation of this propensity quintile is necessary to apply a weighting method intended to 
remove initial differences in student background characteristics. 

Step 2 - Weighting. The purpose behind the creation of the propensity scalar is to 
control for differences in background characteristics across the attendance intensity 
categories. To achieve this goal, we inversely weighted cases relative to their propensity 
outcome so that within each intensity level an equal number of weighted cases resulted in 
each propensity quintile. We also normalized the weighted cases so that the final weighted 
sample was the same size as the original un-weighted sample. Once balance existed among 
student background characteristics across the intensity levels, we could make valid 
comparisons. When balance was lacking for a specific variable, we added extra terms (i.e., 
variable squared or interaction terms) to the HLM ordinal logistic regression described in 
Step 1. We repeated this process until we achieved balance or balance was not achievable. 
The desired result was a sample with no more differences in background than would be 
expected from a randomly controlled design. If a significant relationship between a given 
background variable and attendance intensity was still present after this process, we included 
that variable as a covariate in the final growth model. 
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Chapter IV: 

Student Cohort Demographic Analysis and HLM Modeling Results 

In order to provide more elarity to our analyses, the demographie analysis and 
modeling results will be presented separately by eohort. The synthesis of the results of the 
two eohorts will be presented in the Diseussion & Conelusion seetion. 

Grade 3 Cohort - Student Population Characteristics 

For the Grade 3 eohort, we eondueted student aehievement and demographie analyses 
by subjeet eontents: Math and English-language arts. 

Student achievement. Tables 1 and 2 present standardized CST aehievement means 
for the Grade 3 eohort in math and English-language arts from the 2002-03 sehool year 
through 2005-06 for eaeh intensity eategory. Student aehievement in both math and English- 
language arts was higher for the students with over 100 days of EA’s BEST attendanee in 
eaeh of the 4 years when eompared to students who attended EA’s BEST less often. For 
example, in 2002-03 the standardized CST aehievement mean in math was 0.146 for 
students who averaged over 100 days of EA’s BEST attendanee eompared to a standardized 
CST aehievement mean of -0.107 for students who averaged 20 days or less of LA’s BEST 
attendanee. We tested these differenees by attendanee intensity with four separate one-way 
ANOVA’s (one for eaeh year). The results were statistieally signifieant for eaeh year in both 
math and English-language arts {p < .05). These findings indieate that there were differenees 
in math and English-language arts CST performanee for students with varying levels of LA’s 
BEST attendanee. In addition these differenees exist for eaeh year ineluded in this study. 



Table 1 

Math Achievement by LA’s BEST Attendance Intensity, Grade 3 Cohort 





Average LA’s BEST attendance intensity 
(2003-04 to 2005-06) 


ANOVA 

results 


Unweighted standardized 
math outcome 


1-20 days 
(«= 1,131) 


21-50 days 
(« = 784) 


51-100 days 
(« = 744) 


Over 
100 days 
(«= 1,372) 


A test 


Sig. 


CST math, 2002-03 


-0.107 


-0.078 


-0.014 


0.146 


15.752 


0.000 


CST math, 2003-04 


-0.085 


-0.103 


0.030 


0.115 


11.727 


0.000 


CST math, 2004-05 


-0.120 


-0.115 


-0.010 


0.175 


23.360 


0.000 


CST math, 2005-06 


-0.119 


-0.078 


-0.003 


0.152 


17.558 


0.000 
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Table 2 

English-Language Arts Achievement by LA’s BEST Attendance Intensity, Grade 3 Cohort 



Average LA’s BEST attendance intensity ANOVA 

(2003-04 to 2005-06) results 



Over 



Unweighted standardized 
language arts outcome 


1-20 days 
(«= 1,144) 


21-50 days 
(« = 785) 


51-100 days 
(k = 749) 


100 days 
(«= 1,382) 


E’test 


Sig. 


CST ELA, 2002-03 


-0.099 


-0.089 


0.063 


0.158 


17.144 


0.000 


CST ELA, 2003-04 


-0.408 


-0.415 


-0.307 


-0.176 


14.442 


0.000 


CST ELA, 2004-05 


0.045 


0.049 


0.161 


0.294 


19.309 


0.000 


CST ELA, 2005-06 


0.029 


0.054 


0.149 


0.275 


16.676 


0.000 



Student demographics. Tables 3 and 4 present the student baekground eharaeteristies 
for the Grade 3 eohort for eaeh intensity eategory. Not surprisingly, student attendanee in day 
sehool was assoeiated with the intensity of attendanee in LA’s BEST. Those students with 
higher attendanee intensity in LA’s BEST also attended day sehool more often. Students who 
attended LA’s BEST more frequently were also more likely to be Blaek, female, elassified as 
IFEP, and have parents with more than a high sehool edueation. Students who attended LA’s 
BEST more frequently were also less likely to be Hispanie, or elassified as LEP. All of the 
baekground eharaeteristies presented in the math sample had statistieally signifieant 
differenees aeross the four attendanee intensity eategories {p < .05). In addition, all but one of 
the baekground eharaeteristies presented in the English-language arts sample had statistieally 
signifieant differenees aeross the four attendanee intensity eategories {p < .05). This indieates 
that the students have different eharaeteristies aeross the attendanee intensity levels. In order 
to attribute differenees in aehievement outeomes solely to the level of intensity of 
partieipation it is neeessary to eontrol for these baekground differenees. A propensity seoring 
method was used to reduee the differenees among the groups in an attempt to ereate a final 
sample that would have no signifieant differenees in these eharaeteristies. 
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Table 3 

Background Variables by LA’s BEST Attendance Intensity, Math Sample of the Grade 3 Cohort 



Average LA’s BEST attendance intensity ANOVA 

(2003-04 to 2005-06) results 



Over 



Unweighted standardized 
math outcome 


1-20 days 
(k= 1,131) 


21-50 days 
(k = 784) 


51-100 days 
(« = 744) 


100 days 
(k= 1,372) 


Ftest 


Sig. 


Day school attendance 
(2004-05 to 2005-06) 


153.850 


155.633 


156.156 


158.325 


11.727 


0.000 


Female 


0.478 


0.483 


0.522 


0.582 


23.360 


0.000 


Black 


0.050 


0.046 


0.074 


0.073 


17.558 


0.000 


Hispanic 


0.901 


0.927 


0.872 


0.845 


15.752 


0.000 


IFEP 


0.077 


0.079 


0.095 


0.109 


8.646 


0.000 


LEP 


0.812 


0.786 


0.719 


0.686 


11.198 


0.000 


Parent < HS education 


0.435 


0.418 


0.363 


0.328 


3.600 


0.013 


Parent is HS grad/No college 


0.207 


0.207 


0.220 


0.235 


12.637 


0.000 


Parent had some college 


0.112 


0.125 


0.134 


0.169 


3.238 


0.021 



Table 4 

Background Variables by LA’s BEST Attendance Intensity, English-Language Arts Sample of the Grade 3 Cohort 

Average LA’s BEST attendance intensity ANOVA 

(2003-04 to 2005-06) results 



Unweighted standardized 
math outcome 


1-20 days 
(k= 1,131) 


21-50 days 
(k = 784) 


51-100 days 
(« = 744) 


Over 
100 days 
(k= 1,372) 


Ftest 


Sig. 


Day school attendance 
(2004-05 to 2005-06) 


153.740 


155.625 


156.083 


158.138 


8.336 


0.000 


Female 


0.478 


0.484 


0.521 


0.582 


11.286 


0.000 


Black 


0.050 


0.046 


0.076 


0.072 


3.767 


0.010 


Hispanic 


0.902 


0.927 


0.870 


0.847 


12.653 


0.000 


IFEP 


0.078 


0.079 


0.095 


0.109 


3.126 


0.025 


LEP 


0.809 


0.786 


0.717 


0.687 


20.255 


0.000 


Parent < HS education 


0.433 


0.419 


0.363 


0.326 


12.202 


0.000 


Parent is HS grad/No college 


0.205 


0.206 


0.219 


0.236 


1.477 


0.219 


Parent had some college 


0.112 


0.126 


0.136 


0.171 


6.625 


0.000 
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Controlling for existing population differences. Because students were not randomly 
assigned to the four intensity levels, they displayed different characteristics across the four 
levels of intensity in attendance. Therefore, it was necessary to control for existing student 
background characteristics so that we could explore causal interpretations. In Tables 5 and 6, 
we show the relationship between each background variable and LA’s BEST attendance 
intensity for the sample after we have made adjustments (weighting based on a propensity 
scalar). 

Table 5 

Background Variables by LA’s BEST Attendance Intensity (After Weighting), Math Sample of the Grade 3 
Cohort 



Average LA’s BEST attendance intensity ANOVA 

(2003-04 to 2005-06) results 



Background variables 


1-20 days 
(k= 1,131) 


21-50 days 
(« = 784) 


51-100 days 
(« = 744) 


Over 
100 days 
(« = 1,372) 


T’test 


Sig. 


Zscore: CST math, 2002-03 


-0.011 


0.000 


-0.051 


-0.001 


0.480 


0.696 


Day school attendance 
(2004-05 to 2005-06) 


154.637 


156.126 


155.411 


156.018 


0.968 


0.407 


Female 


0.523 


0.515 


0.508 


0.526 


0.244 


0.866 


Black 


0.068 


0.058 


0.068 


0.053 


1.107 


0.345 


Hispanic 


0.872 


0.910 


0.884 


0.883 


2.329 


0.073 


IFEP 


0.093 


0.091 


0.095 


0.089 


0.082 


0.970 


LEP 


0.758 


0.745 


0.733 


0.759 


0.720 


0.540 


Parent < HS education 


0.388 


0.383 


0.375 


0.382 


0.112 


0.953 


Parent is HS grad/No college 


0.218 


0.211 


0.218 


0.224 


0.173 


0.915 


Parent had some college 


0.134 


0.148 


0.128 


0.139 


0.496 


0.685 



The results in Table 5 demonstrate that, through use of the weighting process, we were 
able to remove nearly all of the bias associated with the relationship between the background 
variables and attendance intensity for the math sample. For example, in the weighted sample 
the percentage of female students across the four attendance categories ranges from a low of 
about 51% (51 to 100 days) to a high of about 53% (over 100 days). The significance test for 
gender has a />-value equal to 0.866, which indicates that these differences are not statistically 
significant. Before the weighting process was applied the percentage of female students 
across the four attendance categories ranged from a low of about 48% (1 to 20 days) to a high 



18 









of about 58% (over 100 days) and these differenees were statistieally signifieant. Other 
examples show that there was only about a .05 standard deviation differenee between the 
high and low CST math mean and a differenee of about a 1.5 attendanee days for the day 
sehool attendanee range. Similar results are seen in the weighted sample for nearly all the 
baekground eharaeteristies. Generally these results allow us to eonelude that there is balanee 
for the baekground variables aeross attendanee intensity eategories in the weighted sample. 
Thus, it would be reasonable to expeet the results after weighting to be eomparable with 
those from a randomly eontrolled design. The relationship between being Hispanie and 
attendanee intensity did approaeh statistieal signifieanee {p > .05). For this reason, Hispanie 
status was ineluded as an additional variable in the final growth model for math. 

Table 6 

Background Variables by LA’s BEST Attendance Intensity (After Weighting), English-Language Arts Sample 
of the Grade 3 Cohort 

Average LA’s BEST attendance intensity ANOVA 







(2003-04 to 2005-06) 




results 


Background variables 


1-20 days 
(«= 1,144) 


21-50 days 
(k = 785) 


51-100 days 
(k = 739) 


Over 
100 days 
(«= 1,382) 


Ftest 


Sig. 


Zscore: CST ELA, 2002-03 


-0.034 


-0.023 


-0.012 


-0.025 


0.072 


0.975 


Day school attendance 
(2004-05 to 2005-06) 


154.396 


156.047 


155.211 


155.780 


0.072 


0.361 


Female 


0.517 


0.517 


0.500 


0.524 


1.069 


0.072 


Black 


0.067 


0.056 


0.068 


0.054 


0.955 


0.413 


Hispanic 


0.873 


0.913 


0.884 


0.882 


2.637 


0.048 


IFEP 


0.090 


0.088 


0.090 


0.091 


0.018 


0.997 


LEP 


0.761 


0.753 


0.745 


0.755 


0.213 


0.887 


Parent < HS education 


0.390 


0.385 


0.384 


0.377 


0.141 


0.935 


Parent is HS grad/No college 


0.219 


0.211 


0.213 


0.224 


0.210 


0.889 


Parent had some college 


0.130 


0.147 


0.126 


0.143 


0.783 


0.503 



As with the math sample, the results in Table 6 show that through use of the weighting 
proeess, we were able to remove most of the bias assoeiated with the relationship between 
the baekground variables and attendanee intensity for the English-language arts sample. The 
relationship between being Hispanie and attendanee intensity was, however, statistieally 
signifieant {p < .05). This indieates that after weighting there were still some differenees 
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across the attendance intensity categories in the proportion of Hispanic students. In addition, 
the relationship between being female and attendance intensity approached statistical 
significance {p > .05). For this reason, we included both Hispanic status and gender as 
additional controlling variables in the final growth model for English-language arts with this 
cohort. 

Three-Level HLM Growth Model Results for Grade 3 Cohort 

We employed a three-level hierarchical growth model to examine the impact of 
afterschool attendance intensity on student achievement. Two separate models were 
conducted for this cohort, one for math and one for English-language arts. 

Math achievement. Table 7 includes the results from the three-level HEM growth 
model for math. We ran this model on the weighted sample that we had already adjusted to 
create balance among the background characteristics. The table presents model effects on 
both the baseline achievement level (intercept) and achievement growth (slope). The P- value 
indicates the statistical significance level of each effect, whereas the unstandardized 
B coefficient indicates the magnitude and direction of the effects. We tested the effect of 
attendance intensity in both EA’s BEST and day school against math achievement at baseline 
(2002-03) and math growth over the course of the study (2002-06). The B coefficient 
indicates that for every year a student maintains regular EA’s BEST attendance (over 100 
days) their math achievement will increase by 0.034 standard deviations relative to a student 
with negligible EA’s BEST attendance (0-20 days). This positive achievement growth was 
statistically significant {p < .05). Interestingly, day school attendance is associated with 
baseline math achievement {p < .05) but not with achievement growth {p > .05). The school- 
level math mean effects on the slope show that students in schools with higher mean math 
achievement at baseline experienced less growth than students in schools with lower baseline 
performance. To be more precise a student from a school which had a baseline math 
performance one standard deviation greater than the mean would experience 0.107 standard 
deviations less growth per year than a student from a school which had average baseline 
math performance. 
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Table 7 

Three-Level HLM Growth Model for Math, Grade 3 Cohort 





B coefficient 


P-value 


Effects on the intercept (Math mean at year 2002-03) 


LA’s BEST attendance over 100 days 


0.010 


0.770 


LA’s BEST attendance 51-100 days 


-0.010 


0.816 


LA’s BEST attendance 21-50 days 


-0.013 


0.785 


Hispanic 


-0.047 


0.493 


Day school attendance 


0.004 


0.001 


School-level math mean, 2002-03 


0.873 


0.000 


Effects on the slope (Math growth from 2002-06) 


LA’s BEST attendance over 100 days 


0.034 


0.001 


LA’s BEST attendance 51-100 days 


0.014 


0.310 


LA’s BEST attendance 21-50 days 


0.004 


0.755 


Hispanic 


-0.008 


0.613 


Day school attendance 


0.000 


0.823 


School-level math mean, 2002-03 


-0.107 


0.013 



Figure 1 displays the estimated aehievement growth trajeetory from baseline for three 
LA’s BEST attendanee intensity eategories in relation to those students who attended LA’s 
BEST on average 0-20 days. The trajectory for students who attended LA’s BEST on 
average 0-20 days is set at zero to serve as a reference line. Relative to students who 
attended LA’s BEST for 20 days or less, students who attended LA’s BEST an average of 
over 100 days saw their predicted math Z-scores grow by just over 0.1 standard deviations 
over the 3 years from baseline. Although this effect size is not large, the growth rate was 
significant {p < .05) and is an important finding given that this effect occurred after we 
carefully controlled background characteristics including day school attendance. Relative to 
students who attended 0-20 days, those who attended 51-100 days appear to experience a 
small degree of positive growth, although the difference was not large enough to reach 
statistical significance {p > .05). 
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Figure 1. Model estimates, Grade 3 cohort - math achievement over time by LA’s BEST attendance 
intensity. 

English-language arts achievement. Table 8 presents the results from the three-level 
HLM growth model for English-language arts. As with the math model, we ran this model on 
the weighted sample that we had already adjusted to ereate balanee among the baekground 
eharaeteristies. Again, we tested the effeet of attendanee intensity in both LA’s BEST and 
day sehool against English-language arts aehievement at baseline (2003-04) and English- 
language arts growth over the course of the study (2003-07). Once again, day school 
attendance is associated with baseline English-language arts achievement {p < .05) but not 
with achievement growth {p > .05). Einlike the math sample results however, LA’s BEST 
attendance intensity was not significantly associated with positive English-language arts 
achievement growth {p > .05). 
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Table 8. Three-Level HLM Growth Model for English-Language Arts, Grade 3 Cohort 





B coefficient 


C-value 


Effects on the intercept (ELA mean at year 2002-03) 


LA’s BEST attendance over 100 days 


-0.024 


0.597 


LA’s BEST attendance 51-100 days 


0.017 


0.688 


LA’s BEST attendance 21-50 days 


-0.001 


0.978 


Female 


0.101 


0.003 


Hispanic 


-0.141 


0.015 


Day school attendance 


0.004 


0.008 


School-level ELA mean, 2002-03 


1.077 


0.000 


Effects on the Slope (ELA growth from 2002-06) 


LA’s BEST attendance over 100 days 


0.020 


0.104 


LA’s BEST attendance 51-100 days 


-0.001 


0.942 


LA’s BEST attendance 21-50 days 


0.005 


0.696 


Female 


0.022 


0.003 


Hispanic 


0.002 


0.874 


Day school attendance 


0.000 


0.694 


School-level ELA mean, 2002-03 


-0.144 


0.000 



Figure 2 displays the expeeted English-language arts aehievement growth over time in 
eaeh LA’s BEST attendanee eategory relative to those students attending less than 20 days. 
The positive growth trend for students who attended LA’s BEST on average over 100 days 
was not large enough to reaeh statistieal signifieanee ip > .05). The other three lines in Figure 
2 are relatively tightly bunehed together eonfirming the finding that there were no signifieant 
differenees between LA’s BEST attendanee intensity and English-language arts aehievement. 
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Figure 2. Model estimates, Grade 3 cohort - English-language arts achievement over time by LA’s BEST 
attendance intensity. 

Next, we present the findings on the Grade 2 cohort. First, we provide the achievement 
analyses, followed with the demographic analyses and the HLM Modeling results. 

Grade 2 Cohort - Student Population Characteristics 

As with the Grade 3 cohort, we conducted student achievement and demographic 
analyses by subject contents: Math and English-language arts. 

Student achievement. Tables 9 and 10 present standardized CST achievement means 
for the Grade 2 cohort in math and English-language arts from the 2003-04 school year 
through the 2006-07 school year for each intensity category. CST achievement scores are not 
equated across time; therefore, the comparisons of interest in these tables are the differences 
within each year across intensity levels and not CST achievement score changes over time. 
Student achievement in both math and English-language arts was highest for the students 
who attended EA’s BEST over 100 days during each of the 4 years. We tested these 
differences by attendance intensity with four separate one-way ANOVA’s for each 
assessment and found them to be significant in each year {p < .05). 
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Table 9. Math Achievement by LA’s BEST Attendance Intensity, Grade 2 Cohort 





Average LA’s BEST attendance intensity 
(2004-05 to 2006-07) 


ANOVA 

results 


Unweighted standardized 
math outcome 


1-20 days 
(k= 1,580) 


21-50 days 
(k= 1,137) 


51-100 days 
(k= 1,213) 


Over 
100 days 
(« = 2,065) 


A test 


Sig. 


CST math, 2003-04 


-0.090 


-0.045 


0.001 


0.043 


5.577 


0.001 


CST math, 2004-05 


-0.056 


0.020 


0.072 


0.117 


10.253 


0.000 


CST math, 2005-06 


-0.096 


-0.062 


0.062 


0.115 


18.554 


0.000 


CST math, 2006-07 


-0.143 


-0.122 


-0.005 


0.045 


11.906 


0.000 



Table 10. English-Language Arts Achievement by LA’s BEST Attendance Intensity, Grade 2 Cohort 



Average LA’s BEST attendance intensity ANOVA 



Unweighted standardized 
language arts outcome 




(2004-05 to 2006-07) 




results 


1-20 days 
(k= 1,576) 


21-50 days 
(k= 1,143) 


51-100 days 
(k= 1,201) 


Over 
100 days 
(k = 2,071) 


Ftest 


Sig. 


CST ELA, 2003-04 


-0.179 


-0.107 


-0.069 


0.030 


13.276 


0.000 


CST ELA, 2004-05 


-0.425 


-0.334 


-0.285 


-0.202 


16.398 


0.000 


CST ELA, 2005-06 


0.099 


0.180 


0.263 


0.325 


16.168 


0.000 


CST ELA, 2006-07 


0.056 


0.122 


0.170 


0.223 


10.662 


0.000 



Student Demographics, Tables 11 and 12 present the student baekground 
eharaeteristies for the Grade 2 eohort for eaeh intensity eategory. Similar to the Grade 3 
eohort, student attendanee in day sehool was assoeiated with the intensity of attendanee in 
the LA’s BEST aftersehool program. Those students with higher attendanee intensity in LA’s 
BEST also attended day sehool more often. Students who attended LA’s BEST more 
frequently were also more likely to be female and have parents with more than a high sehool 
edueation. In addition, students who attended LA’s BEST more frequently were less likely to 
be Hispanie, or elassified as LLP. Most of the baekground eharaeteristies presented in Tables 
11 and 12 had statistieally signifieant differenees aeross the four attendanee intensity 
eategories {p < .05). This indieates that there are substantial existing differenees in student 
baekground eharaeteristies aeross the four attendanee intensity groups. Therefore, it was 
neeessary to eontrol for these differenees in order to draw meaningful inferenees regarding 
the effeet of LA’s BEST attendanee intensity. 
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Table 11. Background Variables by LA’s BEST Attendance Intensity, Math Sample of the Grade 2 Cohort 



Average LA’s BEST attendance intensity ANOVA 

(2004-05 to 2006-07) results 



Background variables 


1-20 days 
(«= 1,580) 


21-50 days 
(k= 1,137) 


51-100 days 
(«= 1,213) 


Over 
100 days 
(« = 2,065) 


Ftest 


Sig. 


Day school attendance 
(2004-05 to 2006-07) 


157.223 


157.253 


158.004 


161.194 


19.522 


0.000 


Female 


0.497 


0.503 


0.516 


0.546 


3.413 


0.017 


Black 


0.060 


0.062 


0.080 


0.079 


2.514 


0.057 


Hispanic 


0.893 


0.881 


0.872 


0.845 


6.728 


0.000 


IFEP 


0.056 


0.069 


0.077 


0.076 


2.192 


0.087 


LEP 


0.690 


0.667 


0.655 


0.590 


14.787 


0.000 


Parent < HS education 


0.358 


0.354 


0.326 


0.285 


9.075 


0.000 


Parent is HS grad/No college 


0.189 


0.175 


0.208 


0.193 


1.384 


0.246 


Parent had some college 


0.106 


0.117 


0.124 


0.169 


11.973 


0.000 



Table 12. Background Variables by LA’s BEST Attendance Intensity, English-Language Arts Sample of the 
Grade 2 Cohort 



Average LA’s BEST attendance intensity 
(2004-05 to 2006-07) 



ANOVA 

results 



Over 



Background variables 


1-20 days 
(«= 1,576) 


21-50 days 
(k= 1,143) 


51-100 days 
(k= 1,201) 


100 days 
(k = 2,071) 


Ftest 


Sig. 


Day school attendance 
(2004-05 to 2006-07) 


156.875 


156.776 


157.778 


160.515 


15.241 


0.000 


Female 


0.500 


0.501 


0.517 


0.553 


4.355 


0.005 


Black 


0.062 


0.061 


0.085 


0.080 


3.108 


0.025 


Hispanic 


0.891 


0.883 


0.868 


0.844 


6.756 


0.000 


IFEP 


0.057 


0.068 


0.077 


0.075 


1.934 


0.122 


LEP 


0.690 


0.669 


0.655 


0.591 


14.789 


0.000 


Parent < HS education 


0.359 


0.355 


0.327 


0.286 


9.194 


0.000 


Parent is HS grad/No college 


0.188 


0.178 


0.206 


0.190 


1.034 


0.376 


Parent had some college 


0.108 


0.115 


0.124 


0.169 


11.857 


0.000 
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Controlling for existing population differences. As with the Grade 3 eohort, we did 
not randomly assign students to the four intensity levels. Furthermore, there were substantial 
existing differenees in student baekground eharaeteristies between the four levels. Therefore, 
it was neeessary to eontrol for existing student baekground eharaeteristies so that we eould 
explore eausal interpretations. In Tables 13 and 14, we show the relationship between eaeh 
baekground variable and LA’s BEST attendanee intensity for the sample after we have made 
adjustments (weighting based on a propensity sealar). 



Table 13. Background Variables by LA’s BEST Attendance Intensity (After Weighting), Math Sample of the 
Grade 2 Cohort 





Average LA’s BEST attendance intensity 
(2004-05 to 2006-07) 


ANOVA 

results 


Background variables 


1-20 days 
(«= 1,580) 


21-50 days 
(k= 1,137) 


51-100 days 
(«= 1,213) 


Over 
100 days 
{n = 2,065) 


Ftest 


Sig. 


Zscore: CST math, 2003-04 


-0.057 


-0.022 


-0.004 


-0.030 


0.672 


0.569 


Day school attendance 
(2004-05 to 2006-07) 


157.790 


157.616 


157.761 


159.993 


6.806 


0.000 


Female 


0.518 


0.518 


0.516 


0.511 


0.077 


0.973 


Black 


0.070 


0.068 


0.081 


0.063 


1.299 


0.273 


Hispanic 


0.879 


0.873 


0.871 


0.871 


0.205 


0.893 


IFEP 


0.060 


0.072 


0.078 


0.068 


1.241 


0.293 


LEP 


0.654 


0.643 


0.648 


0.646 


0.138 


0.937 


Parent < HS education 


0.336 


0.340 


0.324 


0.319 


0.695 


0.555 


Parent is HS grad/No college 


0.190 


0.176 


0.207 


0.195 


1.296 


0.274 


Parent had some college 


0.121 


0.124 


0.124 


0.134 


0.554 


0.646 



27 









Table 14. Background Variables by LA’s BEST Attendance Intensity (After Weighting), English-Language 
Arts Sample of the Grade 2 Cohort 







Average attendance intensity 
(2004-05 to 2006-07) 




ANOVA 

results 


Background variables 


1-20 days 
(«= 1,427) 


21-50 days 
(k = 881) 


51-100 days 
(«= 1,034) 


Over 
100 days 
(«= 1,577) 


Ftest 


Sig. 


Zscore: CST ELA, 2003-04 


-0.113 


-0.063 


-0.067 


-0.092 


0.724 


0.538 


Day school attendance 
(2004-05 to 2006-07) 


157.560 


157.144 


157.478 


159.072 


3.441 


0.016 


Female 


0.517 


0.513 


0.516 


0.523 


0.125 


0.945 


Black 


0.071 


0.064 


0.087 


0.063 


2.516 


0.056 


Hispanic 


0.878 


0.876 


0.866 


0.873 


0.317 


0.813 


IFEP 


0.063 


0.072 


0.079 


0.068 


0.932 


0.424 


LEP 


0.653 


0.647 


0.649 


0.646 


0.072 


0.975 


Parent < HS education 


0.332 


0.337 


0.329 


0.326 


0.156 


0.926 


Parent is HS grad/No college 


0.188 


0.180 


0.203 


0.192 


0.719 


0.541 


Parent had some college 


0.122 


0.123 


0.125 


0.134 


0.501 


0.682 



The results in Tables 13 and 14 demonstrate that, through use of the weighting process, 
we were able to remove most of the bias associated with the relationship between the 
background variables and attendance intensity For example, in the weighted math sample the 
percentage of female students across the four attendance categories ranges from a low of 
about 51% for those who attended over 100 days to a high of about 52% in the other 3 
categories. The significance test for gender has a />-value equal to 0.973, which indicates that 
these differences are not statistically significant. Before the weighting process was applied 
the percentage of female students across the four attendance categories ranged from a low of 
about 50% (1-20 days) to a high of about 55% (over 100 days) and these differences were 
statistically significant. Other examples show that there was only about a 0.03 standard 
deviation difference between the high and low CST math mean, and the percentage of LEP 
students ranged from 64% to 65% across the attendance categories. Similar results are seen 
in the weighted sample for most of the background characteristics. There were, however, still 
significant differences in day school attendance intensity across the four LA’s BEST 
attendance categories {p < .05). This indicates that after weighting there were still some 
differences across the attendance intensity categories in the average number of days attending 
day school. In addition, the relationship between being Black and attendance intensity 
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approached statistical significance in the English-language arts sample {p > .05). For this 
reason, we included both Blaek status and day sehool attendanee as additional eontrolling 
variables in the final growth model for English-language arts with this eohort. 

Three-Level HLM Growth Model Results for Grade 2 Cohort 

As with the Grade 3 eohort, we employed a hierarehieal growth model to examine the 
impaet of aftersehool attendanee intensity on student aehievement. Two separate models 
were eondueted for this eohort, one for math and one for English-language arts. This model 
is performed on the weighted sample in whieh differenees in baekground eharaeteristies and 
the initial aehievement outeome aeross intensity levels have been removed. After weighting 
the sample, there were still differenees in the average number of days attending day sehool 
between the EA’s BEST attendanee eategories. In order to eontrol for these remaining 
differenees, we ineluded day sehool attendanee as a eovariate in the growth models for the 
Grade 2 eohort. 

Math achievement. Table 15 presents the results from the three-level HEM growth 
model for math. As previously mentioned, we ran this model on the weighted sample that we 
had already adjusted to ereate balanee among the baekground eharaeteristies. We tested the 
effeet of attendanee intensity in both EA’s BEST and day sehool against math aehievement at 
baseline (2003-04) and math growth over the eourse of the study (2003-07). 

Regular EA’s BEST attendanee (over 100 days) was again signifieantly assoeiated with 
positive aehievement growth relative to students with EA’s BEST attendanee intensity of 
over 0-20 days per year {p < .05). The interpretation of the B eoeffieient indieates that for 
every year a student maintains regular EA’s BEST attendanee (over 100 days) their math 
aehievement will inerease by 0.029 standard deviations relative to a student with negligible 
LA’s BEST attendanee (0-20 days). Onee again, day sehool attendanee is assoeiated with 
baseline math aehievement {p < .05) but not with aehievement growth {p > .05). 

Sehool-level aehievement means from Grade 2 eohort students is positively assoeiated 
with baseline math aehievement {p < .05) but negatively assoeiated with aehievement growth 
ip < .05). As would be expeeted, this indieates that students in sehools with higher mean 
math aehievement at baseline experieneed less growth than those in sehools with lower mean 
math aehievement at baseline (see Fraenkel & Wallen, 1993). 
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Table 15. Three-Level HLM Growth Model for Math, Grade 2 Cohort 





B coefficient 


C-value 


Effects on the intercept (Math mean at year 2003-04) 


LA’s BEST attendance over 100 days 


0.031 


0.231 


LA’s BEST attendance 51-100 days 


0.065 


0.048 


LA’s BEST attendance 21-50 days 


0.063 


0.092 


Day school attendance 


0.006 


0.000 


School-level math mean, 2002-03 


0.934 


0.000 


Effects on the slope (Math growth from 2003-07) 


LA’s BEST attendance over 100 days 


0.029 


0.001 


LA’s BEST attendance 51-100 days 


0.019 


0.064 


LA’s BEST attendance 21-50 days 


-0.012 


0.273 


Day school attendance 


0.000 


0.538 


School-level math mean, 2003-04 


-0.087 


0.015 



Figure 3 displays the positive assoeiation between LA’s BEST attendanee intensity and 
math aehievement. Although CST aehievement seores are not equated aeross time, the 
growth model allows for relative eomparisons between the four intensity eategories. For this 
reason we display the expeeted aehievement growth over time in eaeh LA’s BEST 
attendanee eategory in relation to those students who attended LA’s BEST on average 0-20 
days. Relative to students who attended LA’s BEST 20 days or less, students who attended 
LA’s BEST an average of over 100 days saw their predieted math Z-seores grow by about 
0.09 standard deviations over the 3 years from baseline {p < .05). The relative growth for 
students who attended LA’s BEST on average 51-100 days was not signifieant {p > .05). The 
negative growth trend for students who attended LA’s BEST on average 21-50 days was also 
not large enough to reaeh statistieal signifieanee {p > .05). 
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Figure 3. Model estimates, Grade 2 cohort - math achievement over time by LA’s BEST attendance intensity. 



English-language arts achievement. Table 16 presents the results from the three-level 
HLM growth model for English-language arts. As with the other models, we ran this model 
on the weighted sample that we had already adjusted to ereate balanee among the baekground 
eharaeteristies. Again, we tested the effeet of attendanee intensity in both LA’s BEST and 
day sehool against English-language arts aehievement at baseline (2003-04) and English- 
language arts growth over the eourse of the study (2003-07). 

Similar to the Grade 3 eohort, LA’s BEST attendanee intensity was not signifieantly 
associated with positive English-language arts achievement growth (p > .05). Furthermore, as 
with the Grade 3 cohort, day school attendance is associated with baseline English-language 
arts achievement (p < .05) but not with achievement growth (p > .05). In other words, 
students in schools with higher mean English-language arts achievement at baseline 
experienced less growth than students in schools with lower baseline performance. 
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Table 16. Three-level HLM Growth Model for English-Language Arts, Grade 2 Cohort 





B coefficient 


P-value 


Effects on the intercept (ELA mean at year 2003-04) 


LA’s BEST attendance over 100 days 


0.027 


0.424 


LA’s BEST attendance 51-100 days 


0.061 


0.088 


LA’s BEST attendance 21-50 days 


0.068 


0.119 


Day school attendance 


0.006 


0.000 


Black 


0.061 


0.191 


School-level ELA mean, 2003-04 


0.975 


0.000 


Effects on the slope (ELA growth from 2003-07) 


LA’s BEST attendance over 100 days 


0.008 


0.380 


LA’s BEST attendance 51-100 days 


0.009 


0.315 


LA’s BEST attendance 21-50 days 


-0.006 


0.496 


Day school attendance 


0.000 


0.111 


Black 


-0.023 


0.093 


School-level ELA mean, 2003-04 


-0.107 


0.000 



Figure 4 displays the expeeted English-language arts growth over time in eaeh LA’s 
BEST attendanee eategory also relative to those students who attended LA’s BEST an 
average of 0-20 days. This figure shows that the aehievement trajeetories for the four 
intensity levels are bunehed tightly together. For example, the expeeted English-language 
arts growth over the study period for students with average LA’s BEST attendanee of over 
100 days was just 0.024 standard deviations above the expeeted growth for those who 
attended LA’s BEST an average of 0-20 days. Furthermore, the figure represents the finding 
that there were no signifieant differenees in aehievement growth due to LA’s BEST 
attendanee intensity {p > .05). 
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Figure 4. Model estimates, Grade 2 cohort - English-language arts achievement over time by LA’s BEST 
attendance intensity. 



In Summary 

Results of the analysis suggest that regular attendanee in the LA’s BEST program (over 
100 days per year) leads to positive math achievement growth when compared to students 
with low attendance in the program. This finding was consistent in two separate cohorts of 
students who we followed over a 4-year period and was statistically significant. The finding 
of positive impact for regular LA’s BEST attendance on math achievement growth was 
present after carefully accounting for existing differences in student background 
characteristics, in addition to important indicators such as students’ initial performance levels 
and their day school attendance over the study period. In contrast, we found that students’ 
achievement growth in English-language arts was not significantly related to the students’ 
intensity of attendance in the LA’s BEST program. This finding was also consistent for both 
cohorts of students represented in this study. 
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Chapter V: 

Discussion and Conclusion 

This study set out to fill a research gap by using rigorous methodology to study the 
effects of “dosage” (intensity of afterschool attendance) on students’ academic outcomes. It 
extends the current literature on the impact of afterschool programs in two key ways: First, 
the analyses explicitly modeled achievement longitudinally for 4 years. Second, the study 
used a large sample of over 10,000 students and took extensive care to apply an advanced 
multilevel propensity matching technique to establish a valid study sample from which we 
could generate valid inferences. 

Implications for Methodology 

Outcome studies of afterschool programs typically are designed to compare participants 
with non-participants based on any program attendance. Consequently, participants may 
attend one day in an afterschool program and still be included in the treatment group. 
Furthermore, non-participants may have been enrolled in other afterschool activities and still 
be included in a control group. As stated in a report by Frankel and Daley (2007), two very 
important issues are ignored by most studies: First, “How did the non-participants spend their 
time afterschool?” and second, “How intensive was the participants’ program attendance?” 
(p. 12). Expanding on Frankel and Daley’s strategy and addressing their concerns, this study 
used statistical strategies to reduce selection bias and confirm their findings on the 
importance of “dosage” for afterschool participants. Similar to their study, this study grouped 
afterschool students by their intensity of attendance into four groups (i.e., 0-20, 21-50, 51- 
100, and over 100 days), and compared the three higher intensity levels against the low 
intensity group, thus addressing their second question concerning intensity of participation. 
Additionally, by comparing the low dosage students to high dosage students, this study 
reduces their fore-mentioned concern on how non-participants spend their out-of-school 
time.'"^ 

Although rare in afterschool studies, it is common in intervention and medical studies 
to examine the effect of different levels of treatment or dosage received and compare groups 
receiving low dosage to groups receiving higher dosage (Imbens, 2000; Leon, Mueller, 
Solomon & Keller, 2001). In this study, we considered it logical to compare the low 
attendance students to those with regular program attendance. The rationale is that because 

LA’s BEST requires 5 days of attendance per week. Based on this rationale, this study considered low dosage 
students unlikely to be simultaneously enrolled in another afterschool program. Despite this, propensity scoring 
and covariance methods were used to remove most of the observable characteristic differences anticipated 
between the low dosage and other intensity groups. 
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these students have demonstrated the intent to reeeive treatment (through enrollment in LA’s 
BEST), they ean be eonsidered to have very similar baekground eharaeteristies to those 
students with regular LA’s BEST attendanee, whieh makes them a superior eontrol group 
than those students who have never attended the program. Furthermore, the seleetion bias 
issues that apply to eontrol students who have never attended the program'^ are likely to be 
greater than for those who have demonstrated some level of need for the program through 
their enrollment. Thus, by eonfming our analyses to students who had some eontaet with the 
LA’s BEST program we removed a potential souree for self-seleetion bias. 

Despite this, we realized that many self-seleetion differenees would still exist among 
the students who partieipated in LA’s BEST at the various intensity levels. Therefore, we 
used propensity seores to balanee the samples and eovariates to eliminate any pre-existing 
differenees. By employing a study design that eompares low attendanee students to high 
attendanee students, and uses propensity seores to weight the existing differenees among the 
four intensity groups, we were able to address most of the seleetion bias issues. 

Implication of Results 

Results of the analysis provide evidenee that regular attendanee in the LA’s BEST 
program (over 100 days per year) leads to positive math aehievement growth when eompared 
to students with low attendanee in the program. This finding was eonsistent in two separate 
eohorts of students whom we followed over a 4-year period and was statistieally signifieant. 
Furthermore, it is important to note that this result is obtained after earefully aeeounting for 
existing differenees in students’ baekground eharaeteristies, so that the most plausible 
explanation of this statistieal differenee is in the intensity of attendanee. 

In eontrast, although the trend of English-language arts aehievement growth is positive 
and follows a developmental pattern similar to math, it is not signifieantly related to the 
students’ intensity of attendanee. This finding was also eonsistent for both eohorts of students 
represented in this study. 

Multilevel longitudinal models are used to model student aeademie aehievement over 
time. The multilevel modeling is statistieally neeessary to aeeount for the nested strueture of 
the data, but also provides a tool with whieh we ean examine important between-sehool 
variation in program implementation. Results from this modeling imply that math 
aehievement growth is higher for sehool sites that initially seored lower at the baseline 



such as having a role model at home who attends to their needs, enrolling in other afterschool activities, being 
tutored by a tutoring agency and so forth 
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period, therefore attendanee in LA’s BEST may have the best potential to benefit the students 
enrolled in those sehools. 

It is also interesting to learn that day sehool attendanee is associated with baseline math 
and English-language arts performance but not with achievement growth. In contrast, regular 
LA’s BEST attendance (over 100 days) is significantly related to achievement growth in 
math. This finding suggests that regular attendance at EA’s BEST can have a positive growth 
effect on student achievement beyond the effect of day school attendance. 

Furthermore, implications from this study highlight that simple indicators of program 
participation are inadequate to capture program effects fully. For a program to have impact 
on students’ achievement, the students need to receive sufficient exposure. Participation level 
would be a better indicator of program effects until the field can find methodologies that 
control the self-selection biases that are inherent and hidden in the non-participants. 
Supporting Frankel and Daley’s finding (2007), this study also found that regular afterschool 
program attendance of at least 100 days per year is necessary to reap the program benefits. 

As shown in Figures 1 through 4, students in the Eevel 1 and 2 intensity groups (0-20 
days and 21-50 days, respectively) show flat or slightly negative growth trends, whereas 
students in the Eevel 3 and 4 intensity groups (51-100 and over 100 days, respectively) 
display positive achievement growth. The figures also illustrate that as afterschool attendance 
intensity increases, achievement growth increases as well, with the Eevel 2 group revealing a 
steeper slope than the Eevel 1 group. The exception is the English-language arts sample of 
the Grade 3 cohort, where the Eevel 1 and 2 intensity groups bunch close together. These 
results indicate that EA’s BEST is capable of making a difference in math achievement 
growth, but students need to have regular attendance to reap the benefits of the program. 

Concerning program implementation, this study found that Hispanics, English 
Language Learners, male students, and students from families with lower parent education 
levels are less likely to have regular attendance (over 100 days). Therefore, LA’s BEST can 
increase the benefits of the program to these students by examining the needs of these 
students and families closely and by offering incentives and program activities that will 
entice their regular attendance. 

Conclusion 

This study sets out to fill a research gap by using rigorous methodology to study the 
effects of “dosage” (intensity of afterschool attendance) on students’ academic outcomes. 
The research tracked approximately 10,000 students for 4 years. We found that students who 
attended EA’s BEST for over 100 days per year showed statistically significant achievement 
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growth in math as compared to students who participated 20 days or less per year. This 
aehievement growth is more evident in sehool sites that seored lower in math at the baseline 
level suggesting that students from sehools that are lower performing gain most from the 
program. In other words, LA’s BEST is serving their targeted population (low performing 
students) as intended. LA’s BEST ean improve their effeetiveness by eneouraging all 
students to partieipate at a minimum of 101 days per year. 
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TECHNICAL APPENDIX 



Step I - HEM Ordinal Logistic Regression 

We employed ordinal logistie regression within a HLM framework to model the 
relationship between student baekground eharaeteristies and the likelihood of a student 
attending the LA’s BEST program at the varying intensity levels. Level 1 (student level) 
indieators inelude eontinuous measures for baseline aehievement, and day sehool attendanee 
and dummy variables for parental edueation (less than high sehool, High sehool graduate no 
eollege, some eollege), ethnieity (Hispanie and Blaek), gender (female), LLP, and IFEP 
status. The four-level ordinal attendanee intensity variable used as the outeome. Eaeh sehool 
represents a Level 2 unit and the average sehool aehievement effeet on the student-level 
intereept is ineluded in the model. An example of the model equation syntax is shown below: 

Level- 1 Model 
Prob[R= 1|B] = P'(1) = P(1) 

Prob[R <= 2|B] = P'(2) = P(l) + P(2) 

Prob[R <= 3|B] = P'(3) = P(l) + P(2) + P(3) 

Prob[R<=4|B] = 1.0 

where 

P(l) = Prob[Y(l) = 1|B]= Probability of being in LA’s BEST (0-20 days) 

P(2) = Prob[Y(2) = 1|B]= Probability of being in LA’s BEST (21-50 days) 

P(3) = Prob[Y(3) = 1|B]= Probability of being in LA’s BEST (51-100 days) 

log[P'(l)/(l - P'(l))] = BO + B1*(FEMALE) + B2*(BLACK) + B3*(HISPANIC) + 
B4*(R_ATTENDANCE) + B5*(LTHS) + B6*(HSGRAD) + B7*(SOME_COLL) + 
B8*(LEP) + B9*(IFEP) + B10*(STUDENT_CST) + B1 1*(R_ATTENDANCE2) 

log[P'(2)/(l - P'(2))] = BO + B1*(FEMALE) + B2*(BLACK) + B3*(HISPANIC) + 
B4*(R_ATTENDANCE) + B5*(LTHS) + B6*(HSGRAD) + B7*(SOME_COLL) + 
B8*(LEP) + B9*(IFEP) + B10*(STUDENT_CST) + B1 1*(R_ATTENDANCE2) + d(2) 

log[P'(3)/(l - P'(3))] = BO + B1*(FEMALE) + B2*(BLACK) + B3*(HISPANIC) + 
B4*(R_ATTENDANCE) + B5*(LTHS) + B6*(HSGRAD) + B7*(SOME_COLL) + 
B8*(LEP) + B9*(IFEP) + B10*(STUDENT_CST) + B1 1*(R_ATTENDANCE2) + d(3) 

Level-2 Model 

(Sehool CST aehievement modeled against student intereept) 

BO = GOO + G01*(SCHOOL_CST) + UO 

B1 = GIO, B2 = G20, B3 = G30, B4 = G40, B5 = G50, B6 = G60 

B7 = G70, B8 = G80, B9 = G90, BIO = GlOO, B1 1 = G1 10 
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Resulting model coefficients were then transformed so that a single propensity scalar 
was created, after which the propensity scalar was divided into quintiles. 



Step 2 - Weighting 

The purpose behind the creation of the propensity scalar was to control for differences 
in background characteristics across the attendance intensity categories. To achieve this goal, 
we inversely weighted the cases relative to their propensity outcome so that within each of 
the intensity levels an equal number of weighted cases resulted in each propensity quintile. 
We also normalized the weighted cases so that the final weighted sample was the same size 
as the original un- weighted sample. 



The following SPSS code is used to accomplish this task: 

** first compute aggregate propensity scalar mean by ‘propensity scalar quintile’ & and the 

LA’s Best Intensity variable *****. 

AGGREGATE 
/OUTFIEE = * 

MODE = ADDVARIABEES 
/BREAK = att intensity pr quintile 
/scalar_mean = MEAN(scalar). 

*** compute temporary weight based on ratio **. 

compute wtl = scalar mean/scalar. 

** Compute aggregate sum of cases in each intensity category **. 

AGGREGATE 
/OUTFIEE = * 

MODE = ADDVARIABEES 
/BREAK = attintensity 
/attendsum = n. 

weight by wtl. 

** Compute aggregate weighted sum of cases in each intensity by quintile category **. 

AGGREGATE 
/OUTFIEE = * 

MODE = ADDVARIABEES 
/BREAK = att intensity pr quintile 
/weight_sum = n. 
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** Compute final normalized weight ***. 
eompute fweight = wtl*((attend_sum/5)/weight_sum). 
weight by fweight. 

** Cheek erosstab to be sure that within eaeh intensity eategory eaeh propensity quintile is 
equally represented in the weighted sample. Also eheek that eaeh intensity eategory weighted 
sample size is unehanged from the raw sample **. 

CROSSTABS 

/TABLES = att intensity BY pr quintile 
/FORMAT = AVALUE TABEES 
/CELLS = COUNT 
/COUNT ROUND CEEE . 

Onee balanee exists among student baekground eharaeteristies aeross intensity levels 
valid eomparisons ean be made. When balanee was laeking for a speeifie variable, we added 
extra terms (variable squared or interaetion terms) to the HEM ordinal logistie regression 
deseribed in Step 1. We repeated this proeess until we aehieved balanee or balanee was not 
possibly aehievable. The desired result was a sample where there would be no more 
differenees in baekground than would be expeeted from a randomly eontrolled design. If a 
signifieant relationship between a given baekground variable and attendanee intensity was 
still present after this proeess we ineluded that variable as a eovariate in the final growth 
model. 

Modeling Achievement Growth - Three-level HLM Growth Model 

We employed a three-level hierarehieal growth model to examine the impaet of 
aftersehool attendanee intensity on student aehievement. In this model, Eevel 1 represents 
time nested within students. For the Grade 3 eohort there are four time points (2002-03 to 
2005-06), with aehievement at eaeh time point serving as the outeome. Similarly, for the 
Grade 2 eohort there are also four time points (2003-04 to 2006-07), with aehievement at 
eaeh time point serving as the outeome. The Eevel 1 intereept is initialized at the first time 
point (2002-03 for the Grade 3 eohort). Eevel 2 aeeounts for student-level effeets. At this 
level, EA’s BEST attendanee intensity is modeled against the Eevel 1 aehievement intereept 
and the aehievement slope over time. We also ineluded day sehool attendanee in the model as 
a student-level eovariate. Eike EA’s BEST attendanee, we modeled day sehool attendanee 
against the Eevel 1 aehievement intereept and the aehievement slope over time. Eevel 3 
aeeounts for sehool-level variation. Sehool-level baseline aehievement in the assessment 
being examined is also modeled against the Eevel 1 aehievement intereept and the 
aehievement slope over time. 
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This model is performed on the weighted sample in whieh differenees in baekground 
eharaeteristies and the initial aehievement outeome aeross intensity levels have been 
removed. Therefore, we did not expeet any effeet of LA’s BEST attendanee intensity on the 
aehievement intereept. The primary relationship of interest is that between attendanee 
intensity and the slope of aehievement growth over time. The presenee of a signifieant 
relationship between attendanee intensity and the slope of aehievement growth over time, 
after eontrolling for day sehool attendanee and other baekground eharaeteristies, would 
provide evidenee of the LA’s BEST program attendanee impaet. 



An example of the model for the Grade 2 eohort is shown below: 



Level- 1 Model (CST aehievement modeled aeross time) 

Y = P0 + P1*(T1ME) + E 
Level-2 Model 

(Student LA’s BEST and day sehool attendanee modeled against CST aehievement intereept 
and slope) 

PO = BOO + B01*(R_ATTENDANCE) + B02*(Hl_lntensity) + B03*(MED_Intensity) + 
B04*(LOW_Intensity) + RO 

PI = BIO + B11*(R_ATTENDANCE) + B12*(Hl_lntensity) + B13*(MED_Intensity) + 
B14*(LOW_lntensity) + R1 



Level-3 Model 

(Sehool CST aehievement modeled against student intereept and slope) 



BOO = GOOO + GOOl(SCHOOLCST) + UOO 

BOl =G010 

B02 = G020 

B03 = G030 

B04 = G040 

BIO = GlOO + GlOl(SCHOOLCST) + UlO 

Bll =G110 

B12 = G120 

B13 = G130 

B14 = G140 



46 




